Fix unassigned ml system shard replicas #1315

b4sjoo · 2023-09-09T01:44:40Z

Description

In previous ml system shard replicas always stays on an arbitrary number and cannot expand automatically, which will cause unassigned shards and yellow cluster problem. This PR can resolve this issue.

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Sicheng Song <sicheng.song@outlook.com>

plugin/src/main/java/org/opensearch/ml/indices/MLIndicesHandler.java

…etting Signed-off-by: Sicheng Song <sicheng.song@outlook.com>

…r.java Co-authored-by: Yaliang Wu <ylwu@amazon.com> Signed-off-by: Sicheng Song <sicheng.song@outlook.com>

plugin/src/main/java/org/opensearch/ml/indices/MLIndicesHandler.java

codecov · 2023-09-09T02:03:45Z

Codecov Report

Merging #1315 (596bb7c) into 2.x (104b1b7) will decrease coverage by 0.20%.
The diff coverage is 21.05%.

@@             Coverage Diff              @@
##                2.x    #1315      +/-   ##
============================================
- Coverage     79.40%   79.20%   -0.20%     
+ Complexity     2247     2245       -2     
============================================
  Files           188      188              
  Lines          9074     9089      +15     
  Branches        903      904       +1     
============================================
- Hits           7205     7199       -6     
- Misses         1461     1481      +20     
- Partials        408      409       +1

Flag	Coverage Δ
ml-commons	`79.20% <21.05%> (-0.20%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
...va/org/opensearch/ml/indices/MLIndicesHandler.java	`59.34% <21.05%> (-10.40%)`	⬇️

... and 2 files with indirect coverage changes

Signed-off-by: Sicheng Song <sicheng.song@outlook.com>

ylwu-amzn

LGTM. Thanks for the change!

* Fix unassigned ml system shard replicas Signed-off-by: Sicheng Song <sicheng.song@outlook.com> * Adjust auto replica settings to keep it consistent with AOS default setting Signed-off-by: Sicheng Song <sicheng.song@outlook.com> * Update plugin/src/main/java/org/opensearch/ml/indices/MLIndicesHandler.java Co-authored-by: Yaliang Wu <ylwu@amazon.com> Signed-off-by: Sicheng Song <sicheng.song@outlook.com> * Modify exception handling Signed-off-by: Sicheng Song <sicheng.song@outlook.com> * Modify exception messages Signed-off-by: Sicheng Song <sicheng.song@outlook.com> * Add response check Signed-off-by: Sicheng Song <sicheng.song@outlook.com> * Add response check and exception handling Signed-off-by: Sicheng Song <sicheng.song@outlook.com> * Keep error message consistent Signed-off-by: Sicheng Song <sicheng.song@outlook.com> * Keep error message consistent Signed-off-by: Sicheng Song <sicheng.song@outlook.com> * Keep error message consistent Signed-off-by: Sicheng Song <sicheng.song@outlook.com> --------- Signed-off-by: Sicheng Song <sicheng.song@outlook.com> Co-authored-by: Yaliang Wu <ylwu@amazon.com> (cherry picked from commit 56976e1)

opensearch-trigger-bot · 2023-09-09T04:45:14Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.x 2.x
# Navigate to the new working tree
cd .worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-1315-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 56976e1078e445f27f2358cf066ec45a9d012a9a
# Push it to GitHub
git push --set-upstream origin backport/backport-1315-to-2.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-1315-to-2.x.

* Fix unassigned ml system shard replicas Signed-off-by: Sicheng Song <sicheng.song@outlook.com> * Adjust auto replica settings to keep it consistent with AOS default setting Signed-off-by: Sicheng Song <sicheng.song@outlook.com> * Update plugin/src/main/java/org/opensearch/ml/indices/MLIndicesHandler.java Co-authored-by: Yaliang Wu <ylwu@amazon.com> Signed-off-by: Sicheng Song <sicheng.song@outlook.com> * Modify exception handling Signed-off-by: Sicheng Song <sicheng.song@outlook.com> * Modify exception messages Signed-off-by: Sicheng Song <sicheng.song@outlook.com> * Add response check Signed-off-by: Sicheng Song <sicheng.song@outlook.com> * Add response check and exception handling Signed-off-by: Sicheng Song <sicheng.song@outlook.com> * Keep error message consistent Signed-off-by: Sicheng Song <sicheng.song@outlook.com> * Keep error message consistent Signed-off-by: Sicheng Song <sicheng.song@outlook.com> * Keep error message consistent Signed-off-by: Sicheng Song <sicheng.song@outlook.com> --------- Signed-off-by: Sicheng Song <sicheng.song@outlook.com> Co-authored-by: Yaliang Wu <ylwu@amazon.com> Signed-off-by: Sicheng Song <sicheng.song@outlook.com>

* Fix unassigned ml system shard replicas Signed-off-by: Sicheng Song <sicheng.song@outlook.com> * Adjust auto replica settings to keep it consistent with AOS default setting Signed-off-by: Sicheng Song <sicheng.song@outlook.com> * Update plugin/src/main/java/org/opensearch/ml/indices/MLIndicesHandler.java Co-authored-by: Yaliang Wu <ylwu@amazon.com> Signed-off-by: Sicheng Song <sicheng.song@outlook.com> * Modify exception handling Signed-off-by: Sicheng Song <sicheng.song@outlook.com> * Modify exception messages Signed-off-by: Sicheng Song <sicheng.song@outlook.com> * Add response check Signed-off-by: Sicheng Song <sicheng.song@outlook.com> * Add response check and exception handling Signed-off-by: Sicheng Song <sicheng.song@outlook.com> * Keep error message consistent Signed-off-by: Sicheng Song <sicheng.song@outlook.com> * Keep error message consistent Signed-off-by: Sicheng Song <sicheng.song@outlook.com> * Keep error message consistent Signed-off-by: Sicheng Song <sicheng.song@outlook.com> --------- Signed-off-by: Sicheng Song <sicheng.song@outlook.com> Co-authored-by: Yaliang Wu <ylwu@amazon.com> (cherry picked from commit 56976e1) Co-authored-by: Sicheng Song <sicheng.song@outlook.com>

* Fix unassigned ml system shard replicas * Adjust auto replica settings to keep it consistent with AOS default setting * Update plugin/src/main/java/org/opensearch/ml/indices/MLIndicesHandler.java * Modify exception handling * Modify exception messages * Add response check * Add response check and exception handling * Keep error message consistent * Keep error message consistent * Keep error message consistent --------- Signed-off-by: Sicheng Song <sicheng.song@outlook.com> Co-authored-by: Yaliang Wu <ylwu@amazon.com>

* Add Auto Release Workflow (#1306) * Add Auto Release Workflow Signed-off-by: Sicheng Song <sicheng.song@outlook.com> * Fix release note address Signed-off-by: Sicheng Song <sicheng.song@outlook.com> --------- Signed-off-by: Sicheng Song <sicheng.song@outlook.com> * Bump aws-encryption-sdk-java to fix CVE-2023-33201 (#1309) Signed-off-by: Sicheng Song <sicheng.song@outlook.com> * Add release note for 2.10.0 release (#1312) * Add release note for 2.10.0 Signed-off-by: Sicheng Song <sicheng.song@outlook.com> * Add CVE fix Signed-off-by: Sicheng Song <sicheng.song@outlook.com> --------- Signed-off-by: Sicheng Song <sicheng.song@outlook.com> * fixing doc link (#1318) * fixing doc link Signed-off-by: Dhrubo Saha <dhrubo@amazon.com> * fixing indentation Signed-off-by: Dhrubo Saha <dhrubo@amazon.com> --------- Signed-off-by: Dhrubo Saha <dhrubo@amazon.com> * Fix unassigned ml system shard replicas (#1315) (#1324) * Fix unassigned ml system shard replicas * Adjust auto replica settings to keep it consistent with AOS default setting * Update plugin/src/main/java/org/opensearch/ml/indices/MLIndicesHandler.java * Modify exception handling * Modify exception messages * Add response check * Add response check and exception handling * Keep error message consistent * Keep error message consistent * Keep error message consistent --------- Signed-off-by: Sicheng Song <sicheng.song@outlook.com> Co-authored-by: Yaliang Wu <ylwu@amazon.com> * Adjust index replicas settings to keep consistent with AOS 2.9 (#1325) Signed-off-by: Sicheng Song <sicheng.song@outlook.com> * Make 2.10 release notes up to date (#1345) Signed-off-by: Sicheng Song <sicheng.song@outlook.com> * fix spelling (#1363) Signed-off-by: Kalyan <kalyan.ben10@live.com> * Add neural search default processor for non OpenAI/Cohere scenario (#1274) * Add neural search default pre/post process function support Signed-off-by: zane-neo <zaniu@amazon.com> * Fix UT failures Signed-off-by: zane-neo <zaniu@amazon.com> * Address PR comment to remove nonJson response case Signed-off-by: zane-neo <zaniu@amazon.com> * Fix low code coverage issue Signed-off-by: zane-neo <zaniu@amazon.com> * fix format issue Signed-off-by: zane-neo <zaniu@amazon.com> * Try to fix classNotFound issue in IT Signed-off-by: zane-neo <zaniu@amazon.com> * revert Try to fix classNotFound issue in IT Signed-off-by: zane-neo <zaniu@amazon.com> * Change gson dependency to compileOnly Signed-off-by: zane-neo <zaniu@amazon.com> * Change default pre/post process function name Signed-off-by: zane-neo <zaniu@amazon.com> * Address code review comments Signed-off-by: zane-neo <zaniu@amazon.com> * Make preprocess function to default Signed-off-by: zane-neo <zaniu@amazon.com> * Remove GsonUtil since there already a single instance in StringUtils Signed-off-by: zane-neo <zaniu@amazon.com> * Fix UT failures Signed-off-by: zane-neo <zaniu@amazon.com> * Address comments Signed-off-by: zane-neo <zaniu@amazon.com> * use import instead of fully qualified name Signed-off-by: zane-neo <zaniu@amazon.com> --------- Signed-off-by: zane-neo <zaniu@amazon.com> --------- Signed-off-by: Sicheng Song <sicheng.song@outlook.com> Signed-off-by: Dhrubo Saha <dhrubo@amazon.com> Signed-off-by: Kalyan <kalyan.ben10@live.com> Signed-off-by: zane-neo <zaniu@amazon.com> Co-authored-by: Sicheng Song <sicheng.song@outlook.com> Co-authored-by: Dhrubo Saha <dhrubo@amazon.com> Co-authored-by: Yaliang Wu <ylwu@amazon.com> Co-authored-by: Kalyan <kalyan.ben10@live.com>

Fix unassigned ml system shard replicas

e3e812f

Signed-off-by: Sicheng Song <sicheng.song@outlook.com>

b4sjoo temporarily deployed to ml-commons-cicd-env September 9, 2023 01:44 — with GitHub Actions Inactive

b4sjoo requested a review from ylwu-amzn September 9, 2023 01:44

b4sjoo had a problem deploying to ml-commons-cicd-env September 9, 2023 01:44 — with GitHub Actions Failure

b4sjoo had a problem deploying to ml-commons-cicd-env September 9, 2023 01:44 — with GitHub Actions Error

b4sjoo requested a review from dhrubo-os September 9, 2023 01:44

ylwu-amzn reviewed Sep 9, 2023

View reviewed changes

plugin/src/main/java/org/opensearch/ml/indices/MLIndicesHandler.java Outdated Show resolved Hide resolved

ylwu-amzn reviewed Sep 9, 2023

View reviewed changes

plugin/src/main/java/org/opensearch/ml/indices/MLIndicesHandler.java Outdated Show resolved Hide resolved

ylwu-amzn reviewed Sep 9, 2023

View reviewed changes

plugin/src/main/java/org/opensearch/ml/indices/MLIndicesHandler.java Outdated Show resolved Hide resolved

Adjust auto replica settings to keep it consistent with AOS default s…

a734dab

…etting Signed-off-by: Sicheng Song <sicheng.song@outlook.com>

b4sjoo temporarily deployed to ml-commons-cicd-env September 9, 2023 01:53 — with GitHub Actions Inactive

Update plugin/src/main/java/org/opensearch/ml/indices/MLIndicesHandle…

1efe2c1

…r.java Co-authored-by: Yaliang Wu <ylwu@amazon.com> Signed-off-by: Sicheng Song <sicheng.song@outlook.com>

b4sjoo had a problem deploying to ml-commons-cicd-env September 9, 2023 01:53 — with GitHub Actions Error

b4sjoo had a problem deploying to ml-commons-cicd-env September 9, 2023 01:53 — with GitHub Actions Failure

b4sjoo had a problem deploying to ml-commons-cicd-env September 9, 2023 01:53 — with GitHub Actions Error

b4sjoo had a problem deploying to ml-commons-cicd-env September 9, 2023 01:53 — with GitHub Actions Failure

dhrubo-os reviewed Sep 9, 2023

View reviewed changes

Modify exception handling

75ca201

Signed-off-by: Sicheng Song <sicheng.song@outlook.com>

b4sjoo temporarily deployed to ml-commons-cicd-env September 9, 2023 02:14 — with GitHub Actions Inactive

b4sjoo had a problem deploying to ml-commons-cicd-env September 9, 2023 02:14 — with GitHub Actions Failure

b4sjoo had a problem deploying to ml-commons-cicd-env September 9, 2023 02:14 — with GitHub Actions Error

Modify exception messages

e0ee0ab

Signed-off-by: Sicheng Song <sicheng.song@outlook.com>

b4sjoo temporarily deployed to ml-commons-cicd-env September 9, 2023 02:18 — with GitHub Actions Inactive

b4sjoo temporarily deployed to ml-commons-cicd-env September 9, 2023 02:41 — with GitHub Actions Inactive

b4sjoo had a problem deploying to ml-commons-cicd-env September 9, 2023 02:41 — with GitHub Actions Error

b4sjoo had a problem deploying to ml-commons-cicd-env September 9, 2023 02:41 — with GitHub Actions Failure

b4sjoo requested a review from ylwu-amzn September 9, 2023 02:42

ylwu-amzn approved these changes Sep 9, 2023

View reviewed changes

dhrubo-os approved these changes Sep 9, 2023

View reviewed changes

b4sjoo temporarily deployed to ml-commons-cicd-env September 9, 2023 03:14 — with GitHub Actions Inactive

b4sjoo had a problem deploying to ml-commons-cicd-env September 9, 2023 03:14 — with GitHub Actions Error

b4sjoo temporarily deployed to ml-commons-cicd-env September 9, 2023 03:14 — with GitHub Actions Inactive

ylwu-amzn merged commit 56976e1 into opensearch-project:2.x Sep 9, 2023
6 of 9 checks passed

b4sjoo added backport 2.x backport 2.9 backport 2.10 labels Sep 9, 2023

opensearch-trigger-bot bot mentioned this pull request Sep 9, 2023

[Backport 2.10] Fix unassigned ml system shard replicas #1316

Merged

opensearch-trigger-bot bot mentioned this pull request Sep 9, 2023

[Backport 2.9] Fix unassigned ml system shard replicas #1317

Merged

b4sjoo added bug Something isn't working and removed backport 2.x labels Sep 9, 2023

b4sjoo deleted the 2.x_index branch September 9, 2023 04:56

b4sjoo mentioned this pull request Sep 12, 2023

[Backport main] Fix unassigned ml system shard replicas (#1315) #1324

Merged

5 tasks

b4sjoo mentioned this pull request Sep 12, 2023

Adjust index replicas settings to keep consistent with AOS 2.9 #1325

Merged

5 tasks

b4sjoo mentioned this pull request Feb 22, 2024

[BUG] BWC tests are failing on neural search when load model api is called #2155

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix unassigned ml system shard replicas #1315

Fix unassigned ml system shard replicas #1315

b4sjoo commented Sep 9, 2023

codecov bot commented Sep 9, 2023 •

edited

Loading

ylwu-amzn left a comment

opensearch-trigger-bot bot commented Sep 9, 2023

Fix unassigned ml system shard replicas #1315

Fix unassigned ml system shard replicas #1315

Conversation

b4sjoo commented Sep 9, 2023

Description

Check List

codecov bot commented Sep 9, 2023 • edited Loading

Codecov Report

ylwu-amzn left a comment

Choose a reason for hiding this comment

opensearch-trigger-bot bot commented Sep 9, 2023

codecov bot commented Sep 9, 2023 •

edited

Loading